fix(protocol): stop internet providers flapping the relay under unreachable-peer backlogs - #420
Open
kivtxs wants to merge 1 commit into
Open
fix(protocol): stop internet providers flapping the relay under unreachable-peer backlogs#420kivtxs wants to merge 1 commit into
kivtxs wants to merge 1 commit into
Conversation
…chable-peer backlogs
An internet-only provider that accumulates undelivered direct messages to
peers that have gone offline gets disconnected by the relay's per-connection
rate limiter and reconnects on a loop ("flapping"). Three compounding defects
drive an unbounded resend rate to unreachable peers:
1. A resend that has to register a fresh ACK (e.g. after an ACK-timeout
re-queue) restarted the backoff ladder at retry_count 0, pinning
delay_for_retry at its 1s floor forever for a never-ACKing recipient.
AckManager::set_retry_count now carries the retry-queue entry's accumulated
count onto the fresh ACK so backoff keeps climbing.
2. An unconfirmed peer that vanished was re-probed every 5s indefinitely. The
confirmation probe now escalates on the same 15s->600s ladder as the welcome
lifecycle and resets on a reachability edge.
3. The Python relay bridge dropped the relay's recipient-keyed DeliveryError
verdict. It now correlates in-flight sends per recipient (a port of the
iOS/Android RecipientInFlightTracker: record before the wire write, resolve
on MessageSent, fail every live in-flight id on DeliveryError, feed
internet_peer_presence offline), so unreachable peers are learned promptly.
All three are always-on and only reduce redundant relay traffic; the default
delivery contract is unchanged.
Adds an opt-in RetryConfig.edge_driven_unreachable_dm (default false) for
deployments whose peers always interact or advertise presence on return (e.g.
a machine-to-machine capability exchange): after a bounded number of probes a
durably-unreachable DM stops being timed-probed and rests in the outbox,
re-driven only on a reachability edge, and the core resend rate is capped by a
token bucket so a large backlog cannot burst past the relay's rate limit.
Default false preserves the documented perpetual-probe behavior exactly, so
every existing native and third-party integration is unaffected.
Plumbed through core config, the uniffi ProtocolConfig and RetryConfig records,
the UDL, all three regenerated bindings (Swift/Kotlin/Python), and the React
Native TypeScript and native layers. Regression tests added for every mechanism
(Rust: set_retry_count carry-forward, confirmation-probe escalation, edge-driven
parking gate, resend rate cap, flag-defaults-off; Python: RecipientInFlightTracker
record/resolve/drain/unrecord and record-before-write ordering). Docs updated in
docs/configuration.md and docs/message-delivery.md; CHANGELOG entry added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An internet-only provider that accumulates undelivered direct messages to peers that have gone offline gets disconnected by the relay's per-connection rate limiter and reconnects on a loop ("flapping", ~1–2×/min). It does not break live request/response — but it is a persistent connection-stability defect that grows with the size of a provider's unreachable-peer backlog, so any long-running provider serving many ephemeral peers hits it over time.
Root causes (three compounding defects, all always-on fixes)
retry_count 0, pinningdelay_for_retryat its 1s floor forever for a never-ACKing recipient.AckManager::set_retry_countnow carries the retry-queue entry's accumulated count (retry_count + 1) onto the fresh ACK, so backoff keeps climbing. This was the dominant contributor.DeliveryError. It now correlates in-flight sends per recipient — a port of the iOS/AndroidRecipientInFlightTracker(record before the wire write, resolve onMessageSent, fail every live in-flight id onDeliveryError, feedinternet_peer_presence(online=false)) — so unreachable peers are learned promptly.These only reduce redundant relay traffic; they restore the SDK's own documented backoff, they don't change the delivery contract.
Opt-in flood control (default off)
Adds
RetryConfig.edge_driven_unreachable_dm(default false). When enabled, a durably-unreachable DM stops being timed-probed after a bounded number of probes and rests in the outbox, re-driven only on a reachability edge (inbound frame / presence-online); a token bucket caps the core resend rate so a large backlog cannot burst past the relay limit. Intended for deployments whose peers always interact or advertise presence on return (e.g. a machine-to-machine capability exchange).Default
falsereproduces the documented perpetual-probe behavior exactly, pinned by a test that fails if the default ever flips — so every existing native and third-party integration is unaffected. Plumbed through core config, the uniffiProtocolConfig/RetryConfigrecords, the UDL, all three regenerated bindings (Swift/Kotlin/Python), and the RN TypeScript + native layers.Tests & docs
Regression test for every mechanism — Rust:
set_retry_countcarry-forward, confirmation-probe escalation, edge-driven parking gate, resend rate cap, flag-defaults-off. Python:RecipientInFlightTrackerrecord/resolve/drain/unrecord, record-before-write ordering, post-wire-failure keeps the entry. Docs updated (docs/configuration.md,docs/message-delivery.md);CHANGELOGentry added.Validation
cargo test --workspace— 2623 pass / 0 fail (incl. doctests)swift test --package-path bindings/react-native/ios) — 252 pass / 0 fail; the iOS bridge typecheck over the excluded native sources (including the regeneratedGenerated/offline_protocol.swift) completes cleangradle :offlineprotocol:testDebugUnitTest, RN Android CI harness) — 443 pass / 0 fail across 40 suitescargo fmt --all --checkclean;cargo clippy --workspaceclean on changed crates (one pre-existingdata_sync.rs:442lint is unrelated to this change)